PATENT APPLICATION 
IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re application of Docket No: A7870 
RaufIZMAILOV,etal. 

Appln. No. : 09/897,495 Group Art Unit: 2662 

Confirmation No.: 2079 Examiner: SabaTSEGAYE 
Filed: July 03, 2001 

For: PATH PROVISIONING FOR SERVICE LEVEL AGREEMENTS IN 
DIFFERENTIATED SERVICE NETWORKS 

DECLARATION UNDER 37 C.F.R. § 1.131 

Commissioner for Patents 

P.O. Box 1450 

Alexandria, VA 22313-1450 

Sir: 

We, Raul Izmailov, Subir Biswas, and Samrat Ganguly, hereby declare and state as 
follows: 

1 . We are the inventors named in the above-captioned U.S. Application No. 
09/987,495, filed July 3, 2001, which claims priority to U.S. Application No. 60/243,731 filed 
October 30, 2000. 

2. At the time we invented the present invention, we were employed by NEC USA, 
INC. (hereinafter "NEC USA") and was employed at the C&C RESEARCH LABORATORIES 
at NEC USA. 

3. Prior to October 19, 2000, the U.S. Filing Date of U.S. Patent No. 6,744,769, the 
invention as described and claimed in the above referenced application was completed at NEC 
USA, as evidenced by the following: 
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4. Prior to October 1 9, 2000, it was standard practice of employees of NEC USA to 
submit their inventions to the Legal Administrator at NEC USA on a CCRL Technical Report, 
after their inventions were approved by their supervisors, and for the invention to be assigned a 
Reference Number. 

5. Prior to October 19, 2000, having earlier conceived the idea as set forth in the 
specification of the above referenced application, the present invention was formally submitted 
to the Legal Administrator at NEC USA in the form of a CCRL Technical Report. In the 
ordinary course of business and in due time, the present invention was assigned Reference 
Number CCRL 1 1 06, and a CCRL Technical Report was prepared, as shown in Exhibit A, a 
copy of which is attached hereto. 

6. Exhibit A discloses an Invention Disclosure Sheet with the CCRL Technical 
Report, which is an 18-page disclosure document prepared by us. Exhibit A includes subject 
matter that supports at least claims 11,12 and 1 4-1 6. These claims of the present application are 
described at least in the following passages of our document in Exhibit A (fcrther support may 
also be found elsewhere in Exhibit A): 

Claim 11 - Pages 8-11 
Claim 12 -Pages 1-4 
Claim 14 -Pages 10-11 
Claim 15 -Pages 2-6 
Claim 16 -Page 4 
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7. The Invention Disclosure Sheet with the CCRL Technical Report was evaluated 
by Mr. Stephen B. Weinstein, Technical Manager and Area Manager, prior to October 1 9, 2000. 
Once the evaluation had been completed, the evaluation was reviewed and approved by Kojiro 
Watanabe, Vice President of NEC USA, prior to October 19, 2000, and then forwarded to us for 
final approval. We signed the approved papers on October 26, 2000. 

8. At the time the subject matter of the present application was invented, it was 
common practice at NEC USA to have its provisional patent applications prepared and filed by 
persons not employed by NEC USA, 

9. In the ordinary course of business and in due time, NEC USA sent a request to 
Sughrue Mion, PLLC, of Washington, DC, USA requesting filing of a provisional application 
and subsequent preparation and filing of a corresponding utility patent applications. The requests 
were sent from Mr. Yoshi Ryujin, Legal Administrator of NEC USA to Mr. Howard L. Bernstein 
of Sughrue Mion, PLLC on Friday, October 27, 2000. A copy of the above request is attached as 
Exhibit B. 

10. In the ordinary course of business and in due course, Sughrue Mion, PLLC filed 
the provisional applications in the U.S. Patent Office and forwarded copies thereof to NEC USA 
on Monday, October 30, 2000. A copy of the letter from Sughrue Mion, PLLC is attached as 
Exhibit C. 

11. In the ordinary course of business NEC USA reviewed and approved the draft 
applications prepared by Sughrue Mion, PLLC, and U.S. Application No. 09/897,495 was 
subsequently filed, properly claiming priority to the above-described provisional application. 
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12. In view of the foregoing, it is clear that we, the named inventor? of the above- 
captioned application, invented the subject matter of the claims prior to the October 19, 2000 
U.S. filing date of U.S. Patent No. 6,744,769, 

We hereby declare further that all statements made herein are of our own knowledge and 
are true and that all statements made on information and belief are believed to be true; and 
further that these statements were made with the knowledge that wiUrul false statements and the 
like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of 
the United States Code, and that such willful felse statements may jeopardize the validity of the 
application or any patent issuing thereon. 
Date: j 
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. Raufl^ttailov A 



Subir Biswas 



Samrat Ganguly 
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Remarks about patentability of Biswas-Oanguly-Izraailov work on 'Path Provisioning for 
Service Level Agreements in Differentiated Services Networks 11 

This work describes high-performance algorithms for provisioning paths supporting 
DiffServ (Differentiated Services) aggregations of traffic requiring different quality of 
service (QoS) treatments. Making DiffServ work for QoS-sensitive traffic is a major 
objective of network operators. NEC will have a market edge for its QoS Manager 
equipment if this equipment is capable of better path provisioning than the QoS manager 
of other manufacturers. 

Although efficient implementations of the proposed algorithms are not yet specified, I 
believe that they could be, realizing inventions that have at least the possibility of being 
significant contributions to competitive NBC products, Moreover, it is important for 
NEC to accumulate a protective portfolio of patents in the critical area of QoS in IP 
networks. For these reasons, despite the inventors' hesitation to propose this material for 
patentabUity study, I believe that a patentability study is justified 
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S. K. Biswas, S.Ganguly and R. Izmailov 

Path Provisioning for Service Level Agreements in Differentiated Services Networks 
Abstract; 

We study the path provisioning as a mechanism to deliver Service Level Agreements in 
IP Differentiated Services networks. The problem of path provisioning is an NP-complete 
problem, so we propose and analyze (by simulations) several algorithms for solving the 
problem. As our simulations demonstrate, a centralized server consistently delivers a 
better performance than a distributed solution. We also show that the performance of one 
of the proposed algorithms* the greedy algorithm with backtracking, can be very close to 
the optimal one, while being computationally feasible. 



Path Provisioning for Service Level Agreements in Differentiated 

Services Networks 



S. K. Biswas, S.Ganguly and R. Izrnailov 
NEC USA C&C Research Laboratories 



1 Abstract 

We study the path provisioning as a mechanism to deliver Service Level Agreements in IP 
Differentiated Services networks. The problem of path provisioning is an NP-complete problem, so 
we propose and analyze (by simulations) several algorithms for solving the problem. As our 
simulations demonstrate, a centralized server consistently delivers a better performance than a 
distributed solution. We also show that the performance of one of the proposed algorithms, the 
greedy algorithm with backtracking, can be very close to the optimal one, while being 
computationally feasible. 



2 Characterization of IP Differentiated Services 

IP Differentiated Service (Diffserv) has been evolving as an efficient and scalable technique for 
providing QoS within the Internet. The primary goal of Diffserv is to move away from per-flow 
signaling (considered to be unscalable because of its per-flow state storage in the core network) for 
achieving end-to-eod QoS guarantees. The per-flow scalability problem is avoided in Diffserv by 
dealing with IP traffic as class-based aggregated flows as opposed to IntServ approach based on 
application level flows. 



2.1 Diffserv Classes 

In a Diffserv network, each IP packet belongs to one of a number of pre-defined classes and is 
given a service quality specified by its Diffserv class. A detailed description of the Diffserv 
architecture can be found in [1]; here we briefly describe only the components of the Diffserv 
model relevant to our study. 

When an IP packet enters a Diffserv domain (a network area implementing Diffserv 
specifications), it is first classified at the edge and then marked with a Diffserv Code Point (DSCP) 
(2) that is used for identifying its traffic class. The DSCP is encoded at the COS/TOS field [2] in 



1 This author is at the Department of Computer Science* Rutgers University and was working in CCRL as 
a summer intern. 
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the IP packet header. Packet classification can be done based on layer-3, layer-4 or any other 
information that can be extracted from the packet. The exact classification process is outside of the 
scope of Diffserv definition and is left open for specific implementations. Once a packet is marked 
with a specific DSCP* the routers within the cote handle the packet based on its Diffserv class. A 
router exercises class-specific queuing, scheduling and routing in order to satisfy the quality 
guarantees that are specified for the individual classes. Each Diffserv domain is free to define the 
scope of its Diffserv classes and mechanisms to provide the differentiated treatment. In other 
words, a packet can be classified and marked differently by the ingress routers of two different 
Diffserv domains. Neighboring Diffserv domains can enter into Service Level Agreement (SLA) 
PI. These agreements could specify class-specific amount of traffic to be sent between the 
domains. Class-specific policing at the ingress and shaping at the egress is used for satisfying the 
SLAs. 

Scheduling guidelines for Diffserv classes are provided by IETF by a concept called Per Hop 
Behavior (PHB) [4]. This document also specifies the PHBs for a number of DiffServ classes, 
IETF is also in the process of standardizing the usage of the DSCP for coding the Diffserv classes. 
Currently, five standard classes, namely. Expedited Forwarding (EF) [4], Assured Forwarding-1 
(AF1), AF2, AF3 and AF4 l$\ are specified. Within each AF class, three subclasses are defined 
based on their packet drop precedences. In addition to these standardized classes, an ISP can define 
non-standard Diffserv classes using locally-significant DSCP values. 



22 End-to-End Service Characterization 

Although the Diffserv standard specifies class-specilic per-hop forwarding behaviors, it does not 
impose any specific end-to-end service characterization for the individual Diffserv classes* This is 
deliberately left up to earners and vendors to customize service profiles and their mapping to 
individual Diffserv classes. 



Diffserv Classes 


Performance Bounds 


Application 


Expedited Forwarding 


Very low delay/jitter 
Very low loss (10*) 


TDM 

Circuit Emulation 


Assured Forwarding -1 


Low delay/jitter 
Low loss (Iff*) 


CD quality audio 
Broadcast quality video 


AF2 


Low delay 
Low loss (KV 4 ) 


IP Telephony (voice and video) 


AF3 


Loss (10 3 ) 


E-Conunerce 
Web Trading 


AF4 


Loss (10*) 


FTP 
Telnet 

Non-critical Web Applications 



Table 1: Requirements and application mapping for the standard Diffserv classes 

Each Diffserv class within an ISP's domain is characterized by an end-to-end loss bound and an 
end-to-end delay bound. For instance, AF1 can be associated with CD^pality audio with a loss 
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upper bound of 10**. Multiple AF classes are distinguished in terms of their individual loss and 
delay guarantees. The EF class can be characterized as one with arbitrarily small loss and queuing 
delay [4], which can be achieved by priority scheduling for EF packets. A possible mapping of 
several Internet applications to standard Diffserv classes is shown in Table 1 . 

Within an ISP domain (both Point of Presence and backbone), class-specific scheduling and 
routing are required for meeting the specified performance bounds of the Diffserv classes. At 
routers, specific scheduling mechanisms has to be in place, based on the class-specific PHBs, 
Expedited Forwarding, for instance, requires preemptive priority scheduling in order to secure Its 
extremely low tolerance to queueing. On the other hand, AF classes can be processed using 
Weighted Fstfr Queuing (WFQ) [7] with appropriate scheduling weights. These scheduling 
mechanisms within a router are assumed to be non-adaptive* In other words, once the scheduling 
for a class on a router interface is set* it does not change with the varying traffic profile of the 
class. 



3 Service Level Agreements 

The most basic Service Level Agreements (SLA) specification [3] is the aggregated amount of 
traffic that would enter through an ingress router* For example, the SLA for aggregated traffic 
from domain- A to domain-B through ingress router h t in Figure U is specified as the rate X. 
Dornain-A estimates this rate from long-term traffic measurements and its own contracts with its 
other neighbor domains* In Figure 1 , traffic flows only for this particular SLA are shown as solid 
arrows. AO other flows are shown as dotted arrows. Note that the SLA parameters are used for 
policing at the ingress routers (e.g. router /]) and shaping/conditioning at the egress routers (eg. 
router £$). 

The SLA description for E^h can be more specific where the portions of traffic X, going to 
different egresses are also specified. The traffic from domain-^ that arrives at domain-5 through 
ingress I u is expected to go to the egress routers £j and E 2 . The corresponding rates X, and %i 
(where fcAt+A*) are specified in the SLA. As we shall explain later, this flow distribution 
information is extremely useftil for Diffserv path provisioning and other traffic engineering 
purposes. However, if domain-A is not able to estimate this fine-grain flow distribution, then SLA 
contract will not have it. In that case, domain-B will have to estimate it from dynamic traffic 
measurement 

SLAs are specified for aggregated traffic and their duration of validity is generally much larger 
than the duration of the end-applications and their corresponding micro-flows. It is important to 
realize that an SLA specification may or may not capture the short-term rate variations that are 
important for path provisioning and traffic engineering. Short-term traffic rate variations have to 
be handled by other mechanisms. 

Note that the class-specific resource partitioning and scheduling are not captured within the SLA 
specification. SLAs only describe the inter-domain traffic profiles. The usage of SLAs for intra- 
domain path provisioning and traffic engineering is decided solely by the local policies within a 
domain. The constraints, of course, are to satisfy the contractual QoS guarantees for the individual 
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SLAs. The next section is devoted to describing our intra-domain algorithms for path provisioning 
for satisfying the SLAs. 

From the SLA information, each ingress router of a domain can compute the estimated volume of 
class-specific traffic from itself to all other egress routers in the same domain. This way, for every 
class, an NxN traffic matrix is created, where N is the number of edge routers in the domain* 
The (ij)th element of the traffic matrix for a class represents the total bandwidth this class uses 
from ingress router i to the egress router j. For example, in Figure 1, for domain-B, the (l ,l)tb 
element is X t , and the (l,2)th element is Xz. We refer to this element as How from f to j. Once the 
traffic matrices are constructed, the next task is to compute the provisioning routes for each non- 
zero element of those matrices. The computed paths are pinned down using either IP source routing 
or a layer-2 mechanism like MPLS. 

For the EF class, the peak rate (X in Figure 1) of SLA specification is used in the traffic matrices. 
For the AF class, however, the token bucket parameters are lumped into a single rate parameter 
that can be used for constructing the traffic matrix. The token bucket parameters may be used for 
computing the equivalent bandwidth of an SLA specification [5]. For example, the rate information 
for every AF SLA in Figure 1 * is specified as {X p ,A t >B) , where X P is the peak bucket rate, X» is the 

mean bucket rate and B is the iraximum burst size). The interpretation of equivalent bandwidth is 
as follows. If an aggregated flow is allocated with its equivalent bandwidth and the flow complies 
with iu SLA token bucket parameters, then the packets of that flow will be guaranteed the QoS 
specified by die flow. Since the equivalent bandwidth captures the loss requirements, the route 
computation algorithms ensure the loss bounds are automatically guaranteed. The delay bounds are 
not considered in this paper. 

Note that the equivalent bandwidth computation is based on local policies and algorithms within 
the domain. We assume that the class-specific resource partitioning, scheduling and AF QoS 
requirements are uniform within the domain and known to all its routers* The discussion of the 
equivalent bandwidth computation algorithms is outside of the scope of this paper. Instead, we 
concentrate on provisioned path routing algorithms that can be used once a traffic matrix is 
constructed using a local algorithm for equivalent bandwidth computation. 



4 Path Provisioning Problem for Differentiated Services Networks 

In addition to PHBs, class-specific routing is used for achieving the end-to-end intra-domain QoS 
for the Diffserv classes. Packets with Diffserv marking can be forwarded using statically 
provisioned end-to-end paths. Class-specific forwarding paths with bandwidth reservation are 
provisioned for every source-destination pair within an ISP's domain. Paths are computed based on 
the static SLAs for individual Diffserv classes. Any significant change in the service level 
agreement causes re-computation of the provisioned paths. Since path provisioning relies on static 
SLAs, it is more likely to be used in the ISPs* core networks where multiple traffic flows produce 
less dynamic aggregated behavior and therefore, reasonably accurate SLA characterization is 
possible: 



4 



) 



Routes for the provisioned paths can be computed centrally by a poIicy/QoS manager like the 
Bandwidth Broker in [6] or by the source nodes in a distributed manner. In the remaining part of 
the paper, we propose and analyze different algorithms for computing routes for provisioned 
Diffserv paths. 

We partition the problem of path provisioning for Diffserv classes into multiple problems, each 
handling its own Diffserv class. The problems are then solved sequentially, starting form the most 
stringent class, EF, to the least demanding one, AF4. First, the provisioned paths for the EF traffic 
matrix are computed and pinned down. The amount of available bandwidth on the links used in 
these paths is correspondingly adjusted (subtracting the amounts reserved for EF from the 
available capacity of links)* Then* the same procedure is repeated for other Diffserv classes (from 
AF1 to AF4). For each traffic class, we have the following formal problem. 

Consider a directed graph G = (N,E) with N nodes and E links. We associate two real numbers 
with each link e of the graph: 

• C t is the link capacity; 

• B< is the available bandwidth of the link, where C e ^B t . 

Each element (traffic flow) of the traffic matrix (SLA matrix) is a triplet (r t s,d)> where 

• s is an ingress node; 

• d is an egress node; 

• r is traffic rate from ingress node s to egress node d 

The triplets are referred to as 71(j>=(/} , s { > 4), where /si v . and K is the total number of triplets 
(non-zero elements of traffic matrix). Some triplets (flows) may be accepted and others rejected 
because there may not be enough bandwidth available to accept all. The accepted flows have to be 
routed while keeping in mind the following three criteria. 

1. We denote by R the number of rejected triplets 7ty"i),...,7fa); the remaining K-R triplets 
are accepted. The ratio R = R/K (we call it flaw blocking rare) should be as low as 
possible. 

2. We denote by V and W the total amounts of bandwidth of accepted flows and all flows, 
respectively: 

The ratio V = V / W (we call it trctffic acceptance rate} should be as high as possible. 

3. We denote by C the amount of network resources used to accommodate the K-R accepted 
flows: 

where hi is the number of hops in the path for the accepted triplet 71(0. The number C 
(we call it hop-bandwidth product) should be as low as possible. 

The performance metrics R » V and C are conflicting targets; since flows have different bandwidth 
requirements, there is usually a choice of accepting fewer "large 1 ' flows at the expense of more 
"small" flows or vice versa. Since accepted bandwidth directly translates to the revenue earned by 
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the ISP, our first priority is to maximize the traffic acceptance rate V by deciding whch flows are 
accepted and which are rejected. The second priority is to route in a way to minimize the hop* 
bandwidth product C. 

Note that splitting [11] of individual flows is an efficient way of balancing network loads. 
Although flow splitting may increase the total bandwidth of accepted traffic flows, ensuring packet 
ordering for individual micro-flows is difficult with splitting. It may require per-packet layer-3 
lookup and hashing at the ingress routers. In this paper* we do not allow flow splitting; for each 
element in a traffic matrix only one ingress-to-egress path is chosen and subsequently pinned down. 

We consider several approaches to the path-provisioning problem. ■ 

In the first approach, the i* edge router first computes the traffic vector to all the other (AM) edge 
routers* This vector corresponds to the i* row of the traffic matrix (SLA information) of the 
domain. If the SLA information is not locally available, it may be necessary for a central SLA 
manager to download the relevant SLA specifications to the i ,b edge router. Then, independently of 
other routers, each edge router computes and pins down the provisioning paths from itself to all 
other edge routers in the domain. 

Since the ingress routers compute and pin down the paths independently, the result of this 
provisioning can be described as follows* 

1 . List all triplets 7X0 in an arbitrary sequence. 

2. If the list of triplets is not exhausted* select the next triplet 7X0; else, stop. 

3. For triplet T(i)=fo s» 4), compute the shortest path p t from s f to di that satisfies the current 
bandwidth availability of the network: B< > r 4 for for each link e in the path This is done by 
pruning all the links e that cannot support the traffic rate i? (for these links, B t < r,) and 
computing the shortest path on the remaining subnetwork. 

4. If such a path p; exists, recompute B € for each link e in the path pt as B e ;-B € -r,\ 

5. Go to step 2. 

We refer to this approach as naive algorithm and denote it as NA. We use this algorithm as a 
benchmark for other algorithms, in order to evaluate the benefits of having a QoS server handling 
path provisioning computations* The performance of NA approximates thai of distributed path 
provisioning, where edge routers select paths for their traffic flows independently of each other and 
the bandwidth availability information is propagated to the routers by QOSPF protocol. 

A QoS server can coordinate the order and manner of path selection. Problems of that nature have 
been addressed previously in the literature. There has been some work done on single path routing 
service in the context of telephone networks and virtual private network; the routing algorithms 
used in these networks depend on specific switching equipment provided by manufacturers. Lin and 
Wang [8] discuss the scenario with single path routing where cose to be minimized is the maximum 
link utilization factor. The authors cast this as a linear programming problem using Lagrangian 
Relaxation for obtaining suboptimal solutions. As pointed out in Frei and Faltings [9.10], the path- 
provisioning problem is a NP -complete and should be attacked by heuristic algorithms. 

In this paper, we consider two heuristic algorithms. First of them is a greedy algorithm which we 
referred to as iterated sorting algorithm (denoted IS). In this algorithm, we sort the triplets in terms 
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of the first field i.e.» the rate. Thus we assume that the triplets 7T0=fa Add satisfy the condition 
r/^r? 5 ... Srjc , where AT is the total number of triplets with non-zero rates (SLA entries). The 
sorted list is then handled in the same manner as in the NA algorithm 

In each step, the IS algorithm picks the largest (in terms of bandwidth required) flow from the list 
of non-rejected flows and attempts to fit it in the network. The algorithm may be suboptima). as 
illustrated in Figure 2. All the links of the network in Figure 2 have available capacity of 10 units, 
and there are only two triplets in the SLA: (6,5,6) and (5,1,4). Provisioning the path (5,2,3,6) for 
the first triplet (6,5,6) blocks the second triplet, (5,1,4)* 

The blocking of the triplet (5,1 ,4) can be prevented by backtracking and reversing the sequence of 
paths to be provisioned: first, accommodate the triplet (5,1,4) with the path (1,2,3,4) and then the 
triplet (6,5,6) is routed along the path (5,7,8,9,6). This is the idea behind our second heuristic 
algorithm (greedy algorithm with backtracking), which we call sequential path shifting and refer to 
as SPS. To describe the algorithm formally, we use the following notations. 

Given a graph G, we define for each triplet 7(0=(n>£fr<&) * wo paths: 

• Ideal shortest path: SPI(i) is the shortest path in G from s 9 to d t such that for all links e in the 
path, C, > n .In other words, SP1(/) is the shortest path in the absence of bandwidth 
reservations of other triplets* 

• Available shortest path; SPA(i) is the shortest path in C from s t to 4 such that for all the links 
e in the path, B m £ n , In other words, SPA(f) is the shortest path in the presence of bandwidth 
reservations of other triplets. 

For any path p carrying the bandwidth reservation r we define the following two operations: Add 
and Del: 

• Add(G,/?,r): for each link e in the path p t the available bandwidth B t is decreased by n B t ;= 
fl r - r . In other words, Add(C,p,r) adjusts the available bandwidth in G in a way that reflects 
the reservation of bandwidth amount r along the path p. 

• Del(G,/>,r); for each link e in the path p. (he available bandwidth B e is increased by r. JJ, 
B t + r . In other words, Del(G,p,r) adjusts the available bandwidth in G in a way that reflects 
the release of bandwidth amount r along the path p. 

As in the IS algorithm, the SPS algorithm uses the list of triplets sorted in terms of the first field 
i.e., the rate. The triplets are then sequentially selected according to this order. For each triplet, a 
decision is made whether to accept it or not. This decision may involve changing the paths for 
already accepted triplets. In order to incorporate this backtracking, we expand the triplets to 
quadruplets by adding an extra bit b. The bit indicates whether the provisioned path for the flow 
can be altered by subsequent flows. In the beginning, the bit is set as TRUE in for all quadruplets 
(the paths for all flows can be altered). 

For i = 1>.~>K> the SPS algorithm sequentially tries to find a path from the ingress to egress for 
the ith quadruplet. In step /, we consider the quadruplet T(r>(r»$,4,£<) for which the path has to 
be computed. Let if, and H] denote the number of hops in SPI(£ and SPA(D> respectively; if 
$PA(i) is not defined (this happens when there is not enough bandwidth in the network to route the 
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i* flow), we set fi] lo infinity. The difference between H] and H i is the number of bops by which 
SPA(i) exceeds the optima] path SPI(Q. Therefore, we define the sub*optimattty cost W(l) as 
-H ( ). This cost represents the amount of additional bandwidth used by the I th flow when 

compared with the optimal path SPI(r). If W(i) is zero (the available path uses the same amount of 
bandwidth as the ideal one), the path SPAO) is accepted for T(i) and the step i is completed. 

Otherwise, if W(i) > 0, the path SP1(Q cannot be used to accommodate the i* flow since for at least 
one of the links e in SPI(r), its available bandwidth B e is smaller than the flow rate n . We denote 
the set of all such links by 0, where Q {e : ee SPI(i);l?, < If there were no other flows in 

the network, all links of SPI(i) would be able to accommodate flow rate r t . However, because of 
paths already provisioned during the previous r-1 steps, some of the links of SPI(r) (namely, those 
belonging to 0 do not have the available bandwidth necessary to support the flow rate rv . 

We denote by M the subset of those already accepted (during the previous i-1 steps) quadruplets 
7T(1),„., Hi-1) for which the following two conditions hold: 

• The bit r } of quadruplet is TRUE. Therefore, the path SPAO) can be altered. 

• All links e in Q belong lo the path SPAO): QczSPA(j). Therefore, if the bandwidth 
reservation for r } of the quadruplet TO) for its path SPAO*) " removed, the available bandwidth 
at each link e in Q increases by j> Since the i* flow requires bandwidth reservation of r£ 
this increase is sufficient for accommodating the t flow using its path SPI(i). 

If the set M is empty and SPA(0 is defined, we accept it as the path for the i* flow (the path is sub- 
optima), but there is no single path we can remove to accommodate it); otherwise, we reject the i* 
flow. 

If Af is not empty, any of its elements can be used to accommodate the i* flow in the following 
way. For each quadruplet Tij) of the set M, its path SPAO) contains all the links in Q. Therefore, 
removing the path SPAO) the network by releasing the bandwidth reserved for TO), permits 
the jth flow to be accommodated with on the path SPI(Q. The removed quadruplet TXj) now has to 
be routed again, and its new altered route may be longer than the one scheduled originally. For 
each quadruplet 7T(/) of this set 0* e Af)> we define the shifting cost S i as —Lj) where L s and 

Ly are calculated in the following way. 

• Lj is the number of bops in the current path P(j) for the quadruplet T(j). 

• is the number of hops in the path P*(j) calculated in <?* for 7\j), where G* is obtained from 

G by first performing the operation TM(G>P(j)>rj) and then the operation Add((/,SPI(iV<) 
(these operations reflect the result of deleting the bandwidth provisioning for T(f) and adding 
the bandwidth provisioning for 7T(i)), If path P*0) does TOt «ist r we set £ } to infinity. 

Denote by S^the minimum of all the shifting costs S/> achieved for somej-m. Depending on what 
is larger, 5™ or W(i) f one of following two actions is taken. 

1. If > the shifting cost of the previously processed quadruplet 1\m) exceeds the sub- 
optimality cost of quadruplet 7ti> Therefore, we accept SPA(0 as the path for 7W) and do not 
change the path for T(m). 
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2* If Smia S W(0» the shifting cost of the previously processed quadruplet 7\m) is smaller than the 
sub-optimality cost of quadruplet 71(0. Therefore, we route the quadruplet TXm) on the path 
P\m) and reset S¥A(m)=P\m) and we route ihe quadruplet 7(0 on the path SPI(f) and reset 
SPA(i>SPI(i)- Also, if the shift of T(m) resulted in a path longer than 2^ (which was the 

number of hops in the path P(j) for the quadruplet T(m) before the shifting)! we change the bit 
bi of the quadruple 7X0 to FALSE This is done to prevent the shifting of SPA(0 by 

subsequent flows 0+1 AO and to simplify the algorithm (if SPA(0 can be shifted by a 

subsequent flow T[k), the altered path for the quadruplet T(m) can be changed again (as well as 
the paths that were shifted by quadruplet Urn) etc.), which leads to state-space explosion). 

5 * Performance of Path Provisioning Algorithms 

In the previous section, we described three path provisioning algorithms: NA, IS and SPS. We 
analyze their performance for two networks under various loading conditions. 

We start with the network as shown in Figure 3: it represents the physical topology of the IP 
Backbone. The Backbone network consists of 12 nodes, and we assume that every link can carry 
10 units of bandwidth* We selected the following three sources and three destinations. 

• Sources: Seattle (1), San Francisco (2) and Los Angeles (3). 

• Destinations: Cambridge (8), New York (9) and Washington, DC (10). 

Each source generated three flows to all three destinations, which created nine flows in total. The 
traffic rate of each flow was randomly distributed on the interval (0,1 Op), where p is a scale 
parameter that is varied form 0 to 1, The average traffic rate of each flow is thus equal to p/2. We 
tested ruse different values of p (namely, p=0.1. pM),2,... t P=0«9) and, for each value of p, we ran 
all three algorithms (NA. IS and SPS) 1 5 times and computed the following. 

• Average traffic acceptance rate JR, defined as the average accepted traffic volume R normalized 
by the total traffic volume (which is 9p/2 in this particular example). 

• Average flow rejection rate V, defined as the average number V of rejected traffic flows 
normalized by the total number of traffic flows (which is 9 in this particular example). 

• Average hop-bandwidth product C, defined as the average hop-bandwidth product of accepted 
flows. 

For this series of experiments, we also used a fourth algorithm, the brute force one (referred to as 
BF). In this algorithm, for each of the nine source-destination pairs, we enumerated all the paths 
that consist of no more than four hops (the restriction of all paths being no longer than four hops 
was introduced only to limit the explosion of the solution space; in fact, in some rare situations, we 
observed path-provisioning solutions involving longer paths). As a result, there are 
F«7 xl lx U x 12x 1 5 x 17x 9 x 10x 1 1 = 2,565,901,800 different ways to select nine paths (four 

hops or less) for all nine flows. If none of F ways provides a solution (solution is defined as a set of 
routes maximizing R and, for the sets with the same R> minimizing Q, one of the nine flows has to 
be blocked (there are nine ways to do it), and the remaining eight flows can be routed to their 
destinations by reduced sets of eight paths. For instance, if the flow from source 2 to destination 10 
is dropped, then none of the 17 paths from for this source-destination pair is used, and the 
remaining eight flows can be routed in F/l 7= 150,935,400 ways. Overall, eight flows can be routed 
in 
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F/7+ F/l 1+ F/11+ F/15+ F/J7+ F/^f F/10+ Ffl 1=2,143,859,850 

ways. If none of these ways provides a solution, iwo flows have to be dropped (there are 72 ways 
to do it), and the remaining seven flows can be routed to their destinations by reduced sets of seven 
paths, and so on. 

While the performance of NA provides a lower bound of performance, the performance of BF 
algorithm provides a useful upper bound. Using this bound, we can judge how close the proposed 
algorithms are to the optimal one. BF is not meant to be used as an actual path provisioning 
algorithm in realistic scenarios; it takes about 12 hours on a Pentium-120 Mh2 to solve the 
problem for just one set of flows. We used BF algorithm in the series of experiments to calculate 
the same performance metrics as for NA, IS and SPS, 

The results of our experiments are shown on Figures 4-6. For small values of p, we observe that 
there is hardly any difference in all three performance metrics. In other words, for small loads, all 
algorithms perform about the same. However, as the average flow requirement reaches 0.3 (and the 
overall load on the system increases), differences in performance emerge. 

In terms of average traffic acceptance rate (which is our primary target performance metrics) there 
is a visible difference between the decentralized NA algorithm, on one side, and the centralized IS, 
SPS and BF algorithms, on the other side (Figure 4). The relative difference among latter 
algorithms is relatively smaller. Within the set of these algorithms, SPS performs better than IS, 
and BF performs better than SPS. The difference between BF and SPS is small, which indicates 
that the performance of SPS is close to that of the optimal BF algorithm* 

In terms of flow blocking rate (which is our secondary performance metric), the relationship 
between NA and other algorithms (IS, SPS and BF) is reversed: NA accepts, on average, more 
flows than other algorithms (Figure 5), The reason for this reversal is the tradeoff between traffic 
acceptance rate and flow blocking rate: IS, SPS and BF accommodate more traffic volume by 
accepting fewer flows with larger bandwidth requirements. 

Finally, in terms of hop-bandwidth product (which is our third performance metric), there is hardly 
any difference between the algorithms (Figure 6). 

Wc also experimented with larger traffic matrices consisting of 16 flows (Figures 7-9) and 25 
flows (Figure 10-12). In all these experiments, only NA* IS and SPS were tested, without BF (even 
with 16 flows, it would take too much lime to compute the optimal allocation). As in the case of 9 
flows, the algorithms have the same performance for lower loads; as the average flow bandwidth 
increases, differences in performance emerge. These differences exhibit the same patterns as in the 
case of 9 flows. Namely, IS and SPS deliver higher traffic acceptance rate (with SPS slightly 
outperforming IS) than that of NA, while the flow blocking rate of IS and SPS are higher than that 
of NA. Also, the hop-bandwidth product for IS and SPS are higher than that of NA. The reason for 
this is the sub-optimality of the paths used to accommodate the additional traffic flows. 

Besides the IP Backbone network, we experimented with the Kyoto University ATM network 
(Figure 13). The network consists of 5 core nodes fully connected by 10 logical links of 1244 Mbps 
capacity (a logical link consists of two parallel physical links of 622Mbps each). The core nodes 
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are connected to the gateway nodes (switches) of 8 domains (departments) with links of 622 Mbps. 
Inside each domain, there are 8 to 1 0 (normal) nodes, interconnected with 155Mbps links. Some of 
these normal nodes have direct connections to the 5 core nodes via 622 Mbps links, bypassing the 
corresponding gateway node. These normal nodes (with direct connections to the core nodes) are 
called bypass nodes, and each domain has exactly one bypass node* Traffic was sent from each 
intra-domain node to every other intra-domain node (there are 64 infra-domain nodes in Kyoto 
University network. The traffic intensity depends on the relationship between the source and 
destination nodes; 50% of the traffic is intra-domain traffic, whereas the other 50% of the traffic 
goes uniformly to the other seven domains. Since the Kyoto University network is not 
symmetrical, this arrangement leads to dissimilar traffic loading on different links of the network. 

The experiments with the resulting 64x64 traffic matrix are shown in Figures 14-16. The relative 
performance of the algorithm follows the same pattern observed in our experiments with IP 
Backbone. In particular, the SPS algorithm delivers the best traffic acceptance rate, followed 
closely by IS. 

As our experiments demonstrate, there is some value in handling path provisioning problem by a 
centralized QoS server. Our simulations also demonstrated that the performance of one of our 
algorithms (greedy algorithm with backtracking) can be very close to the optimal one, while being 
computationally feasible. 



6 Summary and Conclusions 

We considered the path provisioning problem in Diffserv networks. We proposed and analyzed 
several algorithms solving the problem. We identified the greedy algorithm with backtracking 
algorithms as being consistently the best one for the most important performance metric. In future 
work, we plan to study the issue of path provisioning for less precisely defined SLAs, where exact 
breakdown of outgoing traffic is not known. We also plan to address the option of multiple paths 
used for the same traffic flow. 
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Figure 1: Inter-domain Service Level Agreement. 




Figure 2: Sub-optixnality of the greedy algorithm IS. 




Figure 3: Physical topology of IP Backbone, 
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Figure 4: IP Backbone: traffic acceptance rate for 9 flows. 




Figure S: IP Backbone: flow blocking rate for 9 flows. 
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Figure 6: IP Backbone: hop-bandwidth product for 9 flows. 
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Figure 7: IP Backbone: traffic acceptance rate for 16 flows. 




Figure 8: IP Backbone: flow blocking rate for 16 flows. 
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Figure 9: IP Backbone: hop-bandwidth product for 16 flows. 



15 



) |-»-is -m-m -a-sps] 



* 105 



1 






85 

76- 




65 • 


SB 


55 - 


& 


45 




0.1 



0.3 

average flow bandwidth 



i 

0.2 



0l4 



05 



Figure 10: IP Backbone: traffic acceptance rate for 25 flows. 




Figure 1 1 : IP Backbone: flow blocking rate for 25 flows. 
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Figure 12: IP Backbone: hop-bandwidth product for 25 flows. 
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Figure 13: Physical topology of Kyoto University network. 
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